Source Google Analytics Data API: ability to add multiple property ids #30152

roman-yermilov-gl · 2023-09-05T11:56:07Z

What

#24739

How

Property ID is now a list. Set of streams is generated based on each property id

github-actions · 2023-09-05T11:56:28Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-06T20:39:34Z

source-google-analytics-data-api test report (commit `19754052f4`) - ✅

⏲️ Total pipeline duration: 06mn47s

Step	Result
Connector package install	✅
Build source-google-analytics-data-api docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-analytics-data-api/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

lazebnyi · 2023-09-08T19:16:34Z

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_source.py

@@ -43,7 +43,7 @@ def patch_base_class():
 @pytest.fixture
 def config():
    return {
-        "property_id": "108176369",
+        "property_ids": ["108176369"],


can we add unit test with old config and try test backward compatibility in that way

Done in scope of migration

lazebnyi · 2023-09-08T19:17:33Z

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/spec.json

@@ -4,7 +4,7 @@
    "$schema": "https://json-schema.org/draft-07/schema#",
    "title": "Google Analytics (Data API) Spec",
    "type": "object",
-    "required": ["property_id", "date_ranges_start_date"],
+    "required": ["property_ids", "date_ranges_start_date"],


do you test those changes manually for backward compatibility?

I did, but it seems I missed something. Now migration is prepared and everithing is tested

lazebnyi · 2023-09-08T19:19:50Z

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py

+
+            for report in _config["custom_reports"]:
+                invalid_dimensions = set(report["dimensions"]) - dimensions
+                if invalid_dimensions:


please add spaces between condition blocks and add comments for them

lazebnyi · 2023-09-08T19:20:35Z

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py

+                    raise wrong_property_id_error
+
+            if not metadata:
+                return False, "failed to get metadata, over quota, try later"


Error message better start with upper case letter

lazebnyi · 2023-09-08T19:22:54Z

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_streams.py

@@ -39,6 +39,7 @@ def patch_base_class(mocker):

    return {
        "config": {
+            "property_ids": ["496180525"],


if an old config has not property_ids

what happens next?

Figured out in scope of migration

github-actions · 2023-09-12T21:12:11Z

source-google-analytics-data-api test report (commit `c4573aca3d`) - ✅

⏲️ Total pipeline duration: 10mn35s

Step	Result
Connector package install	✅
Build source-google-analytics-data-api docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-analytics-data-api/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

github-actions · 2023-09-13T16:14:00Z

source-google-analytics-data-api test report (commit `a3a4f1f912`) - ❌

⏲️ Total pipeline duration: 46mn31s

Step	Result
Connector package install	✅
Build source-google-analytics-data-api docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-analytics-data-api/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

github-actions · 2023-09-13T17:07:32Z

source-google-analytics-data-api test report (commit `815f1ed462`) - ✅

⏲️ Total pipeline duration: 51mn35s

Step	Result
Connector package install	✅
Build source-google-analytics-data-api docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-google-analytics-data-api/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-google-analytics-data-api test

lazebnyi

Please run tests with old format config

roman-yermilov-gl · 2023-09-13T18:17:12Z

Please run tests with old format config

The tests were run with old config. If format is new, it means migration was applied during CI run

Talalqaddoumi · 2024-01-17T17:41:35Z

I recommend consolidating the data into the same stream rather than creating a stream per table per property. The latter approach proves unhelpful and leads to complications when refreshing the source schema, especially after the addition of numerous properties. Currently, I'm facing difficulties generating and refreshing the source schema in a setup with 160 properties.

Here is an illustration of the challenge:

Furthermore, adopting this approach would necessitate additional transformations in the destination to append the data into a single table.

roman-yermilov-gl self-assigned this Sep 5, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/google-analytics-data-api labels Sep 5, 2023

roman-yermilov-gl force-pushed the ryermilov/source-google-analytics-data-api-multiple-property-id branch from 1b198b1 to e0e2003 Compare September 5, 2023 11:57

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Sep 5, 2023

roman-yermilov-gl force-pushed the ryermilov/source-google-analytics-data-api-multiple-property-id branch from 7858762 to cc5db93 Compare September 5, 2023 17:09

roman-yermilov-gl requested review from lazebnyi and darynaishchenko September 5, 2023 17:54

roman-yermilov-gl linked an issue Sep 5, 2023 that may be closed by this pull request

Source Google Analytics 4 (GA4): Add the ability to select multiple property id when setting up a GA4 connector #24739

Closed

roman-yermilov-gl marked this pull request as ready for review September 5, 2023 17:56

roman-yermilov-gl added checklist-action-run and removed checklist-action-run labels Sep 5, 2023

roman-yermilov-gl force-pushed the ryermilov/source-google-analytics-data-api-multiple-property-id branch 2 times, most recently from 582e75a to 1975405 Compare September 6, 2023 20:30

airbytehq deleted a comment from github-actions bot Sep 6, 2023

lazebnyi requested changes Sep 8, 2023

View reviewed changes

airbytehq deleted a comment from github-actions bot Sep 12, 2023

roman-yermilov-gl requested a review from lazebnyi September 12, 2023 21:24

roman-yermilov-gl added 4 commits September 13, 2023 19:18

Source Google Analytics Data API: ability to add multiple property ids

af49ca9

Source Google Analytics Data API: add config migration and unit test

c462328

fix expected records

cda8df1

resolve conflicts

a3a4f1f

roman-yermilov-gl force-pushed the ryermilov/source-google-analytics-data-api-multiple-property-id branch from 9e0dc49 to a3a4f1f Compare September 13, 2023 15:25

roman-yermilov-gl added 2 commits September 13, 2023 19:26

fix conflicts

260d51a

fix conflicts

815f1ed

roman-yermilov-gl force-pushed the ryermilov/source-google-analytics-data-api-multiple-property-id branch from 4bfe2e9 to 815f1ed Compare September 13, 2023 16:07

lazebnyi requested changes Sep 13, 2023

View reviewed changes

roman-yermilov-gl requested a review from lazebnyi September 13, 2023 18:17

lazebnyi approved these changes Sep 13, 2023

View reviewed changes

roman-yermilov-gl merged commit 6442bd1 into master Sep 14, 2023
26 of 27 checks passed

roman-yermilov-gl deleted the ryermilov/source-google-analytics-data-api-multiple-property-id branch September 14, 2023 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Analytics Data API: ability to add multiple property ids #30152

Source Google Analytics Data API: ability to add multiple property ids #30152

roman-yermilov-gl commented Sep 5, 2023 •

edited

Loading

github-actions bot commented Sep 5, 2023 •

edited by roman-yermilov-gl

Loading

github-actions bot commented Sep 6, 2023

lazebnyi Sep 8, 2023

roman-yermilov-gl Sep 12, 2023

lazebnyi Sep 8, 2023

roman-yermilov-gl Sep 12, 2023

lazebnyi Sep 8, 2023

roman-yermilov-gl Sep 12, 2023

lazebnyi Sep 8, 2023

roman-yermilov-gl Sep 12, 2023

lazebnyi Sep 8, 2023

roman-yermilov-gl Sep 12, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

lazebnyi left a comment

roman-yermilov-gl commented Sep 13, 2023

Talalqaddoumi commented Jan 17, 2024

Source Google Analytics Data API: ability to add multiple property ids #30152

Source Google Analytics Data API: ability to add multiple property ids #30152

Conversation

roman-yermilov-gl commented Sep 5, 2023 • edited Loading

What

How

github-actions bot commented Sep 5, 2023 • edited by roman-yermilov-gl Loading

Before Merging a Connector Pull Request

github-actions bot commented Sep 6, 2023

source-google-analytics-data-api test report (commit 19754052f4) - ✅

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2023

source-google-analytics-data-api test report (commit c4573aca3d) - ✅

github-actions bot commented Sep 13, 2023

source-google-analytics-data-api test report (commit a3a4f1f912) - ❌

github-actions bot commented Sep 13, 2023

source-google-analytics-data-api test report (commit 815f1ed462) - ✅

lazebnyi left a comment

Choose a reason for hiding this comment

roman-yermilov-gl commented Sep 13, 2023

Talalqaddoumi commented Jan 17, 2024

roman-yermilov-gl commented Sep 5, 2023 •

edited

Loading

github-actions bot commented Sep 5, 2023 •

edited by roman-yermilov-gl

Loading

source-google-analytics-data-api test report (commit `19754052f4`) - ✅

source-google-analytics-data-api test report (commit `c4573aca3d`) - ✅

source-google-analytics-data-api test report (commit `a3a4f1f912`) - ❌

source-google-analytics-data-api test report (commit `815f1ed462`) - ✅